In [2]:
from PIL import Image
image = Image.open("../W4/3steps.png")
image
Out[2]:
In [1]:
from fastai import *
from fastai.text import *
import pandas as pd
import numpy as np
import warnings
warnings.filterwarnings('ignore')
In [124]:
#import torch
#defaults.device = torch.device('cpu')
# this is to resolve issue of 'RuntimeError: DataLoader worker is killed by signal' when running learn_clas.lr_find()
Load the data, prepare the language model¶
What is a language model? It has a very specific meaning. A language model is something which predicts next word in a sentence. To predict next word in a sentence you need to know quite a lot about the English language.
https://forums.fast.ai/t/deep-learning-lesson-4-notes/30983
Why is that useful? Because at that point we have got a model that knows how to complete sentences like this. So it knows quite a lot about English and a lot about how the world works. What kinds of things tend to be hot in different situations, for instance. Ideally, it would learn things like in 1996 in a speech to the United Nations, United States President __ said… That will be a really good language model because it needs to know who was the president in that year. Getting really good at training language model is a great way to learn or teach a neural net a lot about what is our world, what’s in our world or how do things work in our world. It’s really a fascinating topic.
In [11]:
dataset = pd.read_csv('../W4/trial.csv', sep = ',')
In [12]:
df = dataset[['overall', 'reviewText']]
In [13]:
df.head()
Out[13]:
In [14]:
df = df.replace({'overall': {1: 'Neg', 2: 'Neg', 4: 'Pos', 5: 'Pos'}})
In [15]:
df2 = df[df['overall']!=3]
In [16]:
len(df2)
Out[16]:
Take unlabelled dataset, use it to fine-tune the pretrained model built with wikitext-103 dataset
Process the unlabelled dataset using TextDataBunch()
In [2]:
path = ''
np.random.seed(42)
bs=48
In [18]:
data_lm = (TextList.from_df(df2, cols ='reviewText')
.split_by_rand_pct(0.2)
.label_for_lm()
.databunch(bs=bs))
In [19]:
data_lm.save('data_lm_export_no3.pkl')
In [20]:
data_lm.show_batch()
Run the above code once, load the data_lm whenever needed
We can then put this in a learner object very easily with a model loaded with the pretrained weights. They'll be downloaded the first time you'll execute the following line and stored in ~/.fastai/models (or a specified path)
In [3]:
data_lm = load_data(path, 'data_lm_export_no3.pkl')
In [4]:
learn = language_model_learner(data_lm, arch=AWD_LSTM, pretrained=True, drop_mult=0.3)
In [23]:
learn.lr_find()
In [24]:
learn.recorder.plot(suggestion = True)
In [25]:
# max_lr will be the 10th of the mininum of the curve = min lss divided by 10: 3.63E-02
learn.fit_one_cycle(1, 4.37e-2, moms=(0.8, 0.7))
In [26]:
learn.save('fit_head_no3')
In [27]:
learn.load('fit_head_no3')
Out[27]:
To complete the fine-tuning, we can then unfreeze and launch a new training
In [29]:
learn.unfreeze()
learn.fit_one_cycle(10, 4.37e-3, moms=(0.8, 0.7))
In [30]:
learn.save('fine_tuned_no3')
In [5]:
learn.load('fine_tuned_no3')
Out[5]:
How good is our model? Well, let's try to see what is predicts after a few given words
How to judge the wellness of the lm(language mode) at this step is to be figured out
In [181]:
TEXT = 'I like the phone case as'
N_WORDS = 40
N_SENTENCES = 3
print("\n".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))
In [185]:
TEXT2 = 'The quality of the phone stand is not very satisfing'
N_WORDS2 = 40
N_SENTENCES2 = 2
In [186]:
print("\n".join(learn.predict(TEXT2, N_WORDS2, temperature=0.20) for _ in range(N_SENTENCES2)))
We have to save the model but also it's encoder, the part that's responsible for creating and updating the hidden state. For the next part, we don't care about the part that tries to guess the next word.
In [36]:
learn.save_encoder('fine_tuned_enc_no3')
Classifier¶
Now we'll create a new data object that only grabs the labelled data and keeps those labels. Again, this line takes a bit of time
In [37]:
df2.to_csv('bi3excluded.csv', sep = ',')
In [38]:
#defaults.cpus=1
path = ''
data_clas = (TextList.from_csv(path, 'bi3excluded.csv', cols = 'reviewText',
vocab=data_lm.vocab)
.split_by_rand_pct(0.3)
.label_from_df(cols='overall')
.databunch(bs=bs))
#.databunch(bs=bs, num_workers=0,))
defaults.cpus = 1, to avoid issue: BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
num_workers = 0, to avoid issue: RuntimeError: DataLoader worker is killed by signal
In [39]:
data_clas.save('data_clas_export_no3.pkl')
In [12]:
data_clas = load_data(path, 'data_clas_export_no3.pkl')
In [13]:
data_clas.show_batch()
In [14]:
learn_clas = text_classifier_learner(data_clas, arch = AWD_LSTM, drop_mult=0.3) # pretrained = True calls for arch=AWD-LSTM
#drop_mult by default 1: 1 will cause underfitting. It's related to regularization,
# regularization might cause underfitting.
In [ ]:
learn_clas.load_encoder('fine_tuned_enc_no3')
In [202]:
learn.freeze()
In [203]:
learn_clas.lr_find()
In [204]:
learn_clas.recorder.plot(suggestion = True, figsize = (9,9))
In [205]:
learn_clas.fit_one_cycle(1, 3.02e-2, moms=(0.8,0.7))
In [206]:
learn_clas.save('first_no3')
In [207]:
learn_clas.load('first_no3')
Out[207]:
In [208]:
learn_clas.freeze_to(-2)
learn_clas.fit_one_cycle(1, slice(2.51e-2/(2.6**4),2.51e-2), moms=(0.8,0.7))
In [209]:
learn_clas.save('second_no3')
In [210]:
learn_clas.load('second_no3')
Out[210]:
In [211]:
learn_clas.freeze_to(-3)
learn_clas.fit_one_cycle(1, slice(2.51e-3/(2.6**4), 5e-3), moms =(0.8, 0.7))
In [212]:
learn_clas.save('third_no3')
In [213]:
learn_clas.load('third_no3')
Out[213]:
In [214]:
learn_clas.unfreeze()
learn_clas.fit_one_cycle(2, slice(2.51e-3/(2.6**4),5e-3), moms=(0.8,0.7))
In [230]:
learn_clas.fit_one_cycle(2, slice(1e-3/(2.6**4),1e-3), moms=(0.8,0.7))
In [215]:
learn_clas.save('final_no3')
In [15]:
learn_clas.load('final_no3')
Out[15]:
In [217]:
learn_clas.show_results()
In [16]:
preds, target = learn_clas.get_preds()
In [17]:
interp = ClassificationInterpretation.from_learner(learn_clas)
In [18]:
interp.plot_confusion_matrix()
In [19]:
#accuracy:
acc = (2628+578)/(2628+578+158+103)
acc
Out[19]:
In [20]:
# sensitivity: true positive detect as true tp/(tp+fn)
sen= (2628/(2628+103))
sen
Out[20]:
In [21]:
#specificity: tn/(tn+fp)
spec = (578/(578+158))
spec
Out[21]:
In [22]:
pre = (2628/(2628+158))
pre
Out[22]:
In [23]:
f1 = 2*(pre*sen)/(pre+sen)
f1
Out[23]:
In [24]:
y_test = list(target.numpy())
y_pred = preds.numpy()[:,1]
In [25]:
import matplotlib.pyplot as plt
from sklearn.metrics import roc_curve, auc
In [26]:
fpr, tpr, thresholds = roc_curve(y_test, y_pred)
#fpr: false positive rate = FP/N = (FP/FP+TN) fall-out/false alarm ratio: rate of negative incorrectly detected as positive
#tpr: true positive rate = TP/P= (TP/TP+FN) = sensitivity, rate of a true positive detected as true
In [27]:
auc_score = auc(fpr, tpr)
In [28]:
plt.plot([0,1], [0, 1], 'k--')
plt.plot(fpr, tpr, label = 'Area = {:.3f}'.format(auc_score))
plt.xlabel('False positive rate')
plt.ylabel('True positive rate')
plt.title('ROC curve')
plt.legend(loc='best')
plt.show()
plt.savefig('binaryno3.png')
In [29]:
import scikitplot as skplt
y_preds_arr = preds.numpy()
skplt.metrics.plot_cumulative_gain(y_test, y_preds_arr)
Out[29]:
In [30]:
skplt.metrics.plot_lift_curve(y_test, y_preds_arr)
Out[30]:
In [31]:
def biggernum(arr):
if arr[0] > arr[1]:
return arr[0]
else:
return arr[1]
In [32]:
def prediction(str):
pred = learn_clas.predict(str)
print('The review belongs to class: ', pred[0], ',', 'with a scale of ', '{:.2f} '.format(biggernum(pred[2].numpy()) * 100), 'out of 100.')
In [ ]:
msg = "This table is complete garbage. The materials are substandard and the overall quality is below what a collage freshman would even expect."
prediction(msg)
In [29]:
msg = "Sturdy enough to serve as a coffee table or a tv stand! I bought this for my friend who moved into a new apartment so that he could have a nice coffee table but since his tv was sitting on the floor he decided to use it as a tv stand instead and it looked great. I MIGHT buy him another one to use for the intended purpose."
prediction(msg)
In [63]:
# this is a 3 star rating product review from Amazon, for a furniture.
learn_clas.predict("Doesn't assemble too well. Parts and studs stick out. The mica breaks or chips easily. Be careful when assembling, the holes have not been drilled properly and you might have some issues.")
Out[63]:
In [65]:
learn_clas.predict( "Sturdy enough to serve as a coffee table or a tv stand! I bought this for my friend who moved into a new apartment so that he could have a nice coffee table but since his tv was sitting on the floor he decided to use it as a tv stand instead and it looked great. I MIGHT buy him another one to use for the intended purpose.")
Out[65]:
In [33]:
def get_score2(string):
score =[]
pred=learn_clas.predict(string)
score.append(round(pred[2].numpy()[1]*100,3))
score.append(round(pred[2].numpy()[0]*100,3))
return score
In [34]:
import plotly.graph_objects as go
import matplotlib.pyplot as plt
In [35]:
def plot_go(string):
colors = ['lightgreen', 'white']
fig = go.Figure(data=[go.Pie(labels = ['Positive', ' '],
values = get_score2(string))])
fig.update_traces(
hoverinfo = 'label+percent', textinfo = 'value' , textfont_size =15,
textposition = 'inside',
marker=dict(colors=colors, line=dict(color = '#001000', width =1)))
fig.update_layout(title = {"text": 'Your Business Performance is {:.2f} out of 100'.format(get_score2(string)[0]),
'y': 0.9, 'x': 0.5,
'xanchor': 'center',
'yanchor': 'top'})
#paper_bgcolor ='lightyellow')
fig.show()
In [36]:
def get_score(string):
pred = learn_clas.predict(string)
scores = [round(biggernum(pred[2].numpy()*100), 2), round(100- biggernum(pred[2].numpy()*100),2)]
return scores
def cat(string):
pred = learn_clas.predict(string)
category = pred[0].obj
if category=='Pos':
pred_cat = ['Positive', 'Negative']
else:
pred_cat = ['Negative', 'Positive']
return pred_cat
def col(string):
if cat(string) ==['Positive', 'Negative']:
colors = ['green', 'red']
else:
colors = ['red','green']
return colors
def title(string):
if cat(string) ==['Positive', 'Negative']:
title = 'Well Done!'
else:
title = 'Ummm... Attention!'
return title
def label_result(string):
labels = []
for i in range(2):
labels.append(cat(string)[i])
labels.append(get_score(string)[i])
labels= np.reshape(labels, (2,2)).tolist()
return labels
def title_color(string):
if cat(string) ==['Positive', 'Negative']:
color = 'green'
else:
color = 'red'
return color
In [37]:
def plot_result(string):
plt.figure(figsize=(8,6))
plt.pie(get_score(string), labels = label_result(string), colors = col(string))
plt.title(title(string), color = title_color(string), size = 30)
my_circle=plt.Circle((0,0), 0.5, color = 'white')
p = plt.gcf()
p.gca().add_artist(my_circle)
plt.show()
In [62]:
def plot_resultgo(string):
fig = go.Figure(data=[go.Pie(labels = label_result(string), values = get_score(string), hole=0.5)])
fig.update_traces(textinfo='label', textfont_size=15,
marker=dict(colors=col(string),))
fig.update_layout(title_text = title(string), titlefont_size =30,titlefont_color = title_color(string),
showlegend=True)
#fig.show()
In [38]:
def plot_resultgo2(string):
fig = go.Figure(data=[go.Pie(labels = label_result(string), values = get_score(string), hole=0.5)])
fig.update_traces(hoverinfo='label', textinfo='label', textfont_size=15,
marker=dict(colors=col(string),))
fig.update_layout(title_text = title(string), titlefont_size =30,titlefont_color = title_color(string),
showlegend=True)
fig.show()
In [126]:
plot_result('not good at all')
In [127]:
plot_result('I love this phone case')
In [128]:
plot_result("Cellulartech Says warranty is in Latin America but not here in USA , so after hours of research , calls etc. trying I get a rep at the so called Samsung-Latin which said their policy is 'YOU GET NO WARRANTY'")
In [130]:
plot_result("It is too short. I thought it would be much longer.")
In [131]:
plot_result("I am at fault because I did not look at the measurement for the length. It is the same size as my regular rolling pin. I would have returned, but I missed the deadline. I will use for my Thanksgiving baking, so I do not know the quality of the item or if I will prefer it to my other rolling pin.")
In [39]:
plot_go("I lost my Anker powerbank recently and had to get a new one, but I didn’t want to buy the same Anker one so I tried this powerbank instead, since the watts are a lot higher than my Anker. Needless to say, it did its job, and lasted maybe longer and charged faster than my Anker! I’m surprised and satisfied with my purchase! One thing that I kinda thought was a downside was that it was pretty heavy, but it isn’t too bad!")
In [40]:
plot_go("This unit only charges my phone (3000 mA) 4.5 times, and it should do at least 8 times. My other power bank (26,800 mA) does charge my phone over 8 times. This should be advertised as a 15,000 mA capacity power bank, and not more.")
In [41]:
plot_go("It seemed like a good charger but I did not use before returning it. I found it to be too heavy to carry with me, so I returned it.")
In [43]:
plot_go("this phone case is cute, but it's not that sturdy...")
In [42]:
get_score("I ordered a necklace over a week ago and it still has not yet arrived")
# it is actually a complaint, so what's the difference between complaint and negative sentiment?
Out[42]:
In [81]:
# same product, 1 star rating review
learn_clas.predict("This table is complete garbage. The materials are substandard and the overall quality is below what a collage freshman would even expect.")
Out[81]:
In [82]:
# 5 star rating for another furniture on Amazon
learn_clas.predict("Sturdy enough to serve as a coffee table or a tv stand! I bought this for my friend who moved into a new apartment so that he could have a nice coffee table but since his tv was sitting on the floor he decided to use it as a tv stand instead and it looked great. I MIGHT buy him another one to use for the intended purpose.")
Out[82]:
In [248]:
plot_go("A little too big for my Pomeranian Jack Russle mix. I wish it was made with one or two more holes in the leather strap. Also I wish there was a “hold down” piece for the remaining leather strap so it doesn’t just hang there uncomfortably. Other than that the quality is good.")
In [249]:
plot_go("It's a very nice looking collar and my dog looks handsome in it, but the leather is worn just about clean through after roughly 6 months of use.")
In [44]:
plot_go("For the price I thought this would be a great quality collar. The leather part is not good.")
In [251]:
plot_go("I have a 80-90 lb lab and this hold up and is really beautiful leather like material !!! We go hiking and in the lake all summer and this hold up wonderfully and is strong and cute !")
In [252]:
plot_go("It's a little more dull brown and not so vibrant as the picture shows, just an observation it still looks great. I'd love for my dog to wear it but it didn't come with any holes punched, it didn't say anywhere that I was suppose to punch them myself...")
In [253]:
plot_go("When I first got this collar I loved it, it was pretty and seemed durable. Seriously after 2 weeks it started peeling and cracking. There is no way that this should have happened after 2 weeks of having it on my pit bull, Pixie. She’s not an overly hyper dog. I had to go out and buy another collar for her. Extremely disappointed.2 people found this helpful")
In [254]:
plot_go("Ordered the tan leather option. Very disappointing. The leather is a different color than what’s pictured, much darker. Doesn’t look at all like what’s shown in the picture.")
In [61]:
plot_go("works great if you like to look at your phone sideways ia a total pain in the ass for a vertical look! can't imagine why anyone would want to view their phone sideways while in the car!")
In [62]:
plot_go("The concept of this phone holder is great but it does not stick to my textured dash. It's not worth the frustration to try and get it to stick. We tried exchanging for the same thing thinking it was defective but the second didn't work either. So we are returning both of them for a refund.")
In [63]:
plot_go("I dislike because if you have it in the Vertical position you cannot plug in to charge your phone, only in horizontal you can charge")
In [64]:
plot_go("Well it would work if when it stands up you could charge it,Other then that it’s fine but i need it to stand and be changed")
In [44]:
(413+1797)/(413+1797+53+48)
Out[44]:
In [45]:
(1797/(1797+53))
Out[45]:
In [46]:
413/(413+48)
Out[46]:
In [47]:
(1797/(1797+48))
Out[47]:
In [49]:
2628/(2628+158)
Out[49]: